Download Hugging Face models into Hugging Face cache #1285
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1285
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 84602c8 with merge base 438ebb1 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
install/install_requirements.sh
Outdated
| @@ -106,3 +106,5 @@ fi | |||
| set -x | |||
| $PIP_EXECUTABLE install evaluate=="0.4.3" lm-eval=="0.4.2" psutil=="6.0.0" | |||
| ) | |||
|
|
|||
| export HF_HUB_ENABLE_HF_TRANSFER=1 | |||
There was a problem hiding this comment.
I don't think this will do anything unless people are running install_requirements.sh via source, which would be unusual.
torchchat/cli/builder.py
Outdated
| @@ -73,7 +74,7 @@ def __post_init__(self): | |||
| or (self.pte_path and Path(self.pte_path).is_file()) | |||
| ): | |||
| raise RuntimeError( | |||
| "need to specified a valid checkpoint path, checkpoint dir, gguf path, DSO path, or PTE path" | |||
| f"need to specified a valid checkpoint path, checkpoint dir, gguf path, DSO path, or PTE path {self.checkpoint_path}" | |||
There was a problem hiding this comment.
as written, this reads like it's suggesting a path. maybe s/PTE path/PTE path, got:/
|
Thanks for taking this on!! It'll make things much easier to adopt PS: You'll wanna rebase this onto main (you can use the update branch button then git pull on your machine) |
vmpuri
force-pushed
the
hf-cache
branch
2 times, most recently
from
0ca3941 to
5bf2315
Compare
|
Fixing @swolchok 's suggestions in follow-up commits. Regarding this:
What's the best way to do this? I've currently implemented this as:
Testing: This should clean up everything in one fell-swoop. We could figure out a "lazy" way of doing this, but I think this is best. |
torchchat/cli/download.py
Outdated
|
|
||
| def get_model_dir(model_config: ModelConfig, models_dir: Optional[Path]) -> Path: | ||
| """ | ||
| Returns the directory where the model artifacts are stored. |
There was a problem hiding this comment.
| Returns the directory where the model artifacts are stored. | |
| Returns the directory where the model artifacts are or would be stored. |
torchchat/cli/download.py
Outdated
| os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" | ||
|
|
||
| # For each model with huggingface distribution path, clean up the old location. | ||
| def _delete_old_hf_models(models_dir: Path): |
There was a problem hiding this comment.
Let's name this more concretely and give more context in the docstring
delete_old_hf_models is ambiguous
|
fyi @lessw2020 @kwen2501 we're going to be leveraging the huggingface-cache now |
|
@vmpuri One important thing to double check is the behavior if someone has a HF model already downloaded from a different project |
Why redownload? Also i recommend updating the description for this PR so it's very clear what we'll do and for how long as well as details on how to change the default the cache location. |
|
Since we're in the process of moving files to a new directory, I'd also like to understand how we plan to save and manage quantized models. There are very few use cases which involve doing inference at full precision and thus we should expect that all users will want to quantize and manage the quantized models. In fact, it likely doesn't make sense to keep the full precision models after quantization, at least in most cases. We shouldn't necessarily solve this problem in this diff but there should be an RFC that covers the whole problem so that we can ensure this diff moves towards the overall solution. Otherwise we may interrupt people's file locations a second time |
True - this was my first thought, but there were 2 main things that make this more complex (and therefore more error prone).
Given that re-downloading the models to the Hugging Face cache is guaranteed to download artifacts to the expected location, I figure the tradeoff between long-term reliability & one-time efficiency makes sense. Re-downloading ~50gb of models would be painful once, but manageable (just ~33 minutes even on a relatively-slow 25 mbps connection). Let me know your thoughts (or if you know a way to recover the snapshot ID and blob IDs from our current download) |
Currently, we download models to a local (~/.torchchat by default). For Hugging Face models, we should download to the Hugging Face cache instead.
As per Hugging Face:
This PR also enables hf_transfer, a production-ready Rust library that speeds up downloads from Hugging Face. In my own testing, the speedup was over 2x:
Testing
Download
Generate
Where
List
Remove
Remove Again (file not present)